Papers with human understanding
Interpreting Language Models with Contrastive Explanations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing explanation methods conflate evidence for various features to predict a token . existing explanation methods are less interpretable for human understanding . |
| Approach: | They propose to explain language models contrastively by looking for salient input tokens that explain why the model predicted one token instead of another. |
| Outcome: | The proposed explanations are better than non-contrastive explanations for language models . they show that contrastive explanations improve simulability for human observers . |
Evaluating Explanation Methods for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) has seen great success during recent years. |
| Approach: | They propose a metric that measures the fidelity of explanation methods on translation tasks . they use an efficient approximation to evaluate several explanation methods . |
| Outcome: | The proposed metric is efficient and can be used on translation tasks. |
Evaluating Readability and Faithfulness of Concept-based Explanations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating concepts from different perspectives lack a unified formalization. |
| Approach: | They propose a formal definition of concepts generalizing to diverse concept-based explanations’ settings and apply it to other types of explanations or tasks. |
| Outcome: | Extensive experimental analysis was carried out to determine the evaluation measures for explanation evaluation measures. |
Challenging the Transformer-based models with a Classical Arabic dataset: Quran and Hadith (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing benchmark datasets have a low readability index which does not reflect real-world complex data. |
| Approach: | They constructed a dataset of Quran-verse and Hadith-teaching pairs by consulting sources of reputable religious experts. |
| Outcome: | The proposed models performed on a binary classification task to identify whether two pieces of CA text convey the same underlying message. |
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language. |
| Approach: | They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show . |
| Outcome: | The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly . |
Trigger-Argument based Explanation for Event Detection (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing works on ED use words or phrases to explain models’ inner mechanisms, but for ED, the event structure is more enlightening clues to explain model behaviors. |
| Approach: | They propose a Trigger-Argument based Explanation method which can utilize event structure knowledge to uncover a faithful interpretation for existing ED models at neuron level. |
| Outcome: | The proposed method can reveal the process by which the model predicts on the large-scale MAVEN and the widely-used ACE 2005 datasets. |
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |